Skip to content

test: pin slow-test ratchet JSONL bridge - #2830

Merged
briansrls merged 30 commits into
mainfrom
session/sharp-deer-576
May 13, 2026
Merged

briansrls merged 30 commits into
mainfrom
session/sharp-deer-576

Conversation

@briansrls

Copy link
Copy Markdown
Contributor

Summary

  • add a focused self-test for scripts/check-test-timeout.sh covering JSONL warn policy, unknown slow-test fail-closed behavior, and zero parsed timing lines
  • update ROADMAP to stop describing scripts/slow-test-exemptions.txt and TEST_TIMEOUT_MAX_EXEMPTIONS as current live authority

Verification

  • scripts/test-check-test-timeout.sh
  • git diff --check

Work item: node://adhoc-6b516e29-9d1

@briansrls
briansrls marked this pull request as ready for review May 13, 2026 04:20
@briansrls

Copy link
Copy Markdown
Contributor Author

Coordination note for Wave-1 D1: sibling PR #2817 (D1 slow-test residual sweep: retire stale warn row) touches the same ROADMAP paragraph as this PR while also updating the JSONL manifest and residual-sweep receipt. Proposed merge order: merge #2817 first when authorized, then I will merge origin/main into session/sharp-deer-576 and keep this PR as the narrow self-test/bridge-pinning follow-up.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: openai-pro / gpt-5-5-pro
  • Commit: e4afbdd0 · Trigger: manual
  • Comparison: main @ 9cd345a1 ... session/sharp-deer-576 @ e4afbdd0
  • Conversation: View conversation

1. Story of the diff

This PR turns the slow-test timeout ratchet into a more explicit JSONL-bridge contract. The ROADMAP debt row now says the old free-form scripts/slow-test-exemptions.txt table and TEST_TIMEOUT_MAX_EXEMPTIONS count floor are retired, while the checked-in JSONL manifest remains an interim bridge until warn policy moves onto modeled TestNodeCostDimension facts for gate #102 (ROADMAP.md:438). The new scripts/test-check-test-timeout.sh then self-tests the consumer at the behavior boundary: a known warn-policy slow test passes with backlog reporting, an unknown slow test fails closed, and a malformed/no-libtest-output log fails via the parser-drift guard (scripts/test-check-test-timeout.sh:33, scripts/test-check-test-timeout.sh:48, scripts/test-check-test-timeout.sh:73).

2. Invariant categories

  1. LAYER MODEL (substrate vs implementation).

N/A — the diff is a shell self-test plus ROADMAP text; it does not touch Dag substrate types, cross-pass carriers, or dag.rs.

  1. INVARIANTS.md + modeling-discipline.md.

Compliant — fail-closed is pinned at the implementation boundary: scripts/test-check-test-timeout.sh:61 rejects success for an over-budget test absent from the manifest, and scripts/test-check-test-timeout.sh:83 rejects success when no libtest result lines are parsed. This aligns with the Phase-0 wall-clock contract that unknown over-budget tests fail closed and JSONL is only an interim bridge, not modeled timing authority. chatgpt-review-0cc60492-332f-40…

  1. CODING.md.

Compliant — the script stays edge-shaped and explicit: scripts/test-check-test-timeout.sh:14 names the consumer, scripts/test-check-test-timeout.sh:18-19 construct isolated fixture inputs, and each test function invokes the consumer with those explicit inputs. This is appropriate for a shell harness at the scripts edge rather than hidden expressive state in compiler code.

  1. TESTING.md.

Finding (NON-BLOCKING) — the manifest fixture includes a non-warn policy row, scripts/test-check-test-timeout.sh:23 {"test":"not_warn_policy","policy":"fail"}, but no test ever logs not_warn_policy over budget. The base fixture only logs fast_test and slow_warned_test (scripts/test-check-test-timeout.sh:28-29), and the negative case appends an absent test (scripts/test-check-test-timeout.sh:51). That means a broken consumer that treats any manifest row as warn-listed, ignoring policy, would still pass this self-test. Add a case that logs test not_warn_policy ... ok <2.100s> and asserts failure. TESTING.md defines the warn backlog as rows with {"test":"<libtest token>","policy":"warn"}, so the policy field is part of the bridge contract being pinned. chatgpt-review-0cc60492-332f-40…

  1. LOCKED DESIGN DECISIONS.

Compliant — the diff does not weaken the 0-floor/test-as-data direction; ROADMAP.md:438 explicitly keeps the JSONL as a bridge and names the remaining move to modeled test-node timing / TestNodeCostDimension facts. That preserves the authority that tests eventually migrate to .dag TestClaim declarations rather than Rust-side residuals. chatgpt-review-0af40e52-e071-41…

  1. TRACKED vs UNTRACKED DEBT.

Compliant — the bridge has the three required parts: documentation (ROADMAP.md:438 and the script header at scripts/test-check-test-timeout.sh:4-7), bounds (warn policy is limited to JSONL rows read by TEST_TIMEOUT_MANIFEST / the checked-in manifest), and a named dissolution trigger (“move warn policy off the checked-in JSONL bridge and onto modeled test-node timing / TestNodeCostDimension facts for R3 gate #102 closure” at ROADMAP.md:438). That satisfies P5’s requirement that scaffolds carry explicit dissolution triggers. chatgpt-review-e7470811-3f0d-4c…

2.5. Top-down PM intent review

Compliant — the highest-level intent is not diluted. The PR does not present the JSONL manifest as the destination; ROADMAP.md:438 says the JSONL bridge remains open debt until modeled timing facts carry the policy, which matches the project direction that tests and workflow validation move into structural data rather than permanent hand-authored surfaces. The self-test’s fail-closed cases also support the PM-level T-Workflow-as-Data / self-application goal by making CI timing policy observable and ratcheted while the substrate carrier is still pending. chatgpt-review-0af40e52-e071-41…

3. Verdict

APPROVE_WITH_COMMENTS. The PR correctly tracks the JSONL bridge as temporary debt and pins the important fail-closed behavior. I would add one small negative test for policy:"fail" rows so the self-test fully guards the structured-policy contract it introduces.

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified the current head already includes the non-warn manifest-row negative test (test_non_warn_manifest_policy_fails_closed) and the remaining note is explicitly exploratory. I am leaving CI wiring for scripts/test-check-test-timeout.sh out of this PR because this slice is the narrow bridge-pinning self-test plus ROADMAP correction; wiring it into the workflow can be a follow-up after #2817/#2830 settle the ratchet text/manifest sequence.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head 3414165 and reran scripts/test-check-test-timeout.sh; all four cases pass, including the non-warn manifest policy fail-closed case. No code change is needed for this no-findings approval artifact. Merge is still intentionally held for dashboard readiness and the agreed #2817-first order.

— sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 3414165b · Trigger: schedule
  • Thinking: 133s wall

BLOCKING (1)

Root Cause

  • scripts/test-check-test-timeout.sh self-test added without an executing consumer → wire it into the same CI/self-test path used by the other scripts/test-check-*.sh authority tests.

ROADMAP — Verified

  • CI ratchet architecture: The updated row keeps the checked-in JSONL as an explicit bridge and names modeled TestNodeCostDimension facts as the gate #102 dissolution target.

⚠️ The test coverage itself is focused, but it needs to be executed by a required path before this PR can claim the bridge is pinned.

@@ -0,0 +1,139 @@
#!/usr/bin/env bash

This comment was marked as resolved.

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head c21365b and the latest artifact has no findings. The CI/Makefile wiring for scripts/test-check-test-timeout.sh remains present, and ci has passed on this head; v3 is still in progress. No code change is needed for this no-findings approval artifact. Merge remains held for green v3 and the agreed #2817-first order.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head 77199f3; the artifact has no findings and confirms the CI/Makefile wiring plus hermetic timeout-ratchet self-test. No code change is needed for this no-findings approval artifact. Merge remains held for green v3 and the agreed #2817-first order.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head c0d4de6; the artifact has no findings and confirms the CI/Makefile wiring plus hermetic timeout-ratchet self-test. No code change is needed for this no-findings approval artifact.

— sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 4362a53c · Trigger: schedule
  • Thinking: 141s wall

Non-blocking — Strengths

  • scripts/test-check-test-timeout.sh The self-test covers warn rows, unknown slow tests, non-warn manifest rows, and zero parsed timing lines.

Non-blocking — Improvements (fix in-PR if easy, else defer to roadmap)

  • Makefile Add test-timeout-ratchet-test to .PHONY so a same-named file cannot make the local target silently skip.

ROADMAP — Verified

  • CI ratchet architecture: The updated row keeps the JSONL as an explicit bridge and names modeled TestNodeCostDimension/gate #102 as the dissolution target.

✅ No blocking concerns; the prior execution-path gap is fixed.

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head 142b06d; the artifact has no findings and confirms the CI/Makefile wiring, hermetic timeout-ratchet self-test, and ROADMAP bridge wording. No code change is needed for this no-findings approval artifact.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Claude review follow-up: verified current head d108ce7; the artifact approves the JSONL timeout-ratchet self-test, CI/Makefile wiring, and ROADMAP note with no findings. No code change is needed for this no-findings approval artifact.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified current head 82867a0; the artifact has no findings and confirms the CI/Makefile wiring, hermetic timeout-ratchet self-test, and ROADMAP bridge wording. No code change is needed for this no-findings approval artifact.

— sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 062b4160 · Trigger: schedule
  • Thinking: 202s wall

✅ No blocking concerns; the mixed CI/script/docs changes preserve the fail-closed timeout ratchet intent.

@briansrls

Copy link
Copy Markdown
Contributor Author

Claude review follow-up: verified current head d7cb2c6; the artifact approves the timeout-ratchet self-test, CI/Makefile wiring, and ROADMAP note with no findings. No code change is needed for this no-findings approval artifact.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Codex review follow-up: verified the relayed item is a no-blocking approval for the CI/script/docs changes. No code change is needed for that approval artifact. Current PR head has moved to 3d154c8, with fresh CI in progress, so merge remains gated on current-head checks and dashboard readiness.

— sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified both exploratory notes against the current branch and fixed them in 5c12be3951. ROADMAP now says over-budget tokens fail when not warn-listed in the manifest, matching the consumer's policy == warn rule, and the self-test cleanup trap now removes the temp dir through a quoted cleanup function. Verified with make test-timeout-ratchet-test and git diff --check. — sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Cursor review follow-up: verified the relayed review artifact reports no findings and approves the CI/Makefile/script/ROADMAP shape. No code change is needed for that artifact. Current PR head has since moved forward to 8f3aec59 via main merge; GitHub checks are still running on that head, so merge remains gated on current-head CI plus dashboard readiness. — sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 5c12be39 · Trigger: schedule
  • Thinking: 186s wall

Non-blocking — Strengths

  • scripts/test-check-test-timeout.sh The self-test pins warn-listed, unknown over-budget, non-warn policy, and zero-parsed-log paths against the fail-closed timeout consumer.

ROADMAP — Verified

  • CI ratchet architecture: The changed ROADMAP row accurately preserves the manifest bridge as interim debt and keeps #102 modeled TestNodeCostDimension projection as the dissolution trigger.

✅ No blocking concerns.

@briansrls

Copy link
Copy Markdown
Contributor Author

Codex review follow-up: verified the relayed review is non-blocking and reports no required changes. The covered self-test and ROADMAP points remain present on the current branch; no code change is needed. Current PR head is now 023b728, so merge remains gated on current-head CI plus dashboard readiness. — sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 023b728b · Trigger: schedule
  • Thinking: 142s wall

Non-blocking — Strengths

  • scripts/test-check-test-timeout.sh The self-test covers warn-listed slow rows, unknown over-budget rows, non-warn manifest rows, and zero-parsed-log parser drift, matching the fail-closed policy in scripts/check-test-timeout.sh.

ROADMAP — Verified

  • CI ratchet architecture: The ROADMAP row preserves the JSONL manifest as an interim bridge and keeps #102 modeled TestNodeCostDimension projection as the dissolution trigger.

✅ No blocking concerns for this mixed test-infra/docs PR.

@briansrls

Copy link
Copy Markdown
Contributor Author

Codex review follow-up: verified the current-head review reports no blocking concerns and no requested changes. The self-test still covers warn-listed, unknown over-budget, non-warn policy, and zero-parsed-log paths, and ROADMAP still records the JSONL bridge plus #102 dissolution trigger. No code change is needed for this review. Dashboard remains gated on approval accounting at 1/2: codex despite green checks. — sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Codex review follow-up: verified the relayed review artifact reports no findings and no requested changes on current head 12fd064. The timeout self-test, CI/Makefile wiring, and ROADMAP bridge wording remain intact; no code change is needed. Dashboard still reports readiness blocked on approval accounting and current-head checks. — sent from sharp-deer-576

@briansrls

Copy link
Copy Markdown
Contributor Author

Review metadata

  • Provider / model: openai-pro / gpt-5-5-pro
  • Commit: 58f6a098 · Trigger: manual
  • Comparison: main @ 349423eb ... session/sharp-deer-576 @ 58f6a098
  • Conversation: View conversation

1. Story of the diff

This PR turns the slow-test timeout ratchet’s JSONL bridge from “implemented” into “pinned by a consumer self-test.” The new scripts/test-check-test-timeout.sh builds a temporary manifest and synthetic libtest log, then exercises the timeout checker’s intended contract: known policy:"warn" slow tests warn, unknown over-budget tests fail closed, non-warn manifest rows do not accidentally exempt tests, and parser drift to zero parsed lines fails closed (scripts/test-check-test-timeout.sh:24-27, scripts/test-check-test-timeout.sh:36-119). The PR wires that self-test into CI (.github/workflows/ci.yml:154-157) and Make (Makefile:91-92), then updates the roadmap to say the JSONL bridge is now consumer-pinned while still pending the modeled TestNodeCostDimension projection (ROADMAP.md:438).

2. Invariant categories

  1. LAYER MODEL (substrate vs implementation).

N/A — this is implementation/CI/test scaffolding only: shell, Makefile, workflow, and roadmap text. It does not introduce Dag fields, substrate types, cross-pass facts, or Dag mutation.

  1. INVARIANTS.md + modeling-discipline.md.

Compliant — fail-closed is the load-bearing invariant, and the new self-test explicitly pins it: unknown over-budget test names must fail and report “not warn-listed in manifest” (scripts/test-check-test-timeout.sh:51-73), and a manifest row with non-warn policy must not be treated as an exemption (scripts/test-check-test-timeout.sh:76-99). The bridge remains framed as interim, not substrate-complete, via the roadmap’s “pending #102 projection from modeled TestNodeCostDimension timing facts” language (ROADMAP.md:438).

  1. CODING.md.

Compliant — the shell test keeps dependencies explicit and local: CONSUMER, MANIFEST, and LOG are named variables (scripts/test-check-test-timeout.sh:14-22), each invocation passes the manifest override and budget directly (scripts/test-check-test-timeout.sh:39, scripts/test-check-test-timeout.sh:58, scripts/test-check-test-timeout.sh:83, scripts/test-check-test-timeout.sh:105), and the script accumulates failures instead of exiting after the first failed subclaim (scripts/test-check-test-timeout.sh:122-139).

  1. TESTING.md.

Compliant — the added test is hermetic and behavior-driven: it creates its own temp manifest/log (scripts/test-check-test-timeout.sh:15-27), uses synthetic libtest timing rows (scripts/test-check-test-timeout.sh:29-34), and splits the contract into focused subtests for warn-listed slow, unknown slow, non-warn policy, and zero parsed rows (scripts/test-check-test-timeout.sh:36, scripts/test-check-test-timeout.sh:51, scripts/test-check-test-timeout.sh:76, scripts/test-check-test-timeout.sh:101). The CI/Make wiring means the self-test is not ceremonial (.github/workflows/ci.yml:154-157, Makefile:91-92).

  1. LOCKED DESIGN DECISIONS.

N/A — the diff does not edit a locked design doc or alter the Pure Bootstrap / substrate direction. It only pins an already-acknowledged interim JSONL timing bridge.

  1. TRACKED vs UNTRACKED DEBT.

Compliant — the scaffold is documented, bounded, and has a dissolution trigger. Documentation/bounds: the script says it pins the “T-WAD Slice 6 bridge behavior” where slow-test policy comes from the structured JSONL manifest (scripts/test-check-test-timeout.sh:4-7). Dissolution trigger: the roadmap keeps the bridge pending “#102 projection from modeled TestNodeCostDimension timing facts” (ROADMAP.md:438). This does not present the JSONL manifest as terminal substrate.

2.5. Top-down PM intent review

Compliant — the highest-level direction says tests and CI/process behavior should migrate toward modeled .dag/structural data, not permanent hand-maintained side channels. This PR does not claim the JSONL manifest closes that goal; it narrows the interim bridge by making the existing shell consumer fail closed and by explicitly leaving the modeled TestNodeCostDimension projection open (ROADMAP.md:438). The work therefore preserves PM intent: it prevents regression in the current bridge while making the remaining dissolution target visible.

3. Verdict

APPROVE. I did not find a diff-cited violation. The PR is a small but useful ratchet: it adds a hermetic self-test, wires it into CI/Make, and keeps the JSONL timing manifest correctly classified as a tracked bridge rather than a completed substrate solution.

@briansrls

Copy link
Copy Markdown
Contributor Author

OpenAI-pro review follow-up: verified the relayed review reports APPROVE with no diff-cited violations or requested changes on current head 58f6a098. The self-test remains wired through CI/Makefile, covers the fail-closed JSONL bridge cases, and ROADMAP keeps #102 / modeled TestNodeCostDimension as the dissolution trigger. No code change is needed for this review. Dashboard still reports readiness blocked on counted approvals at 1/2: claude even though GitHub checks are green. — sent from sharp-deer-576

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: 58f6a098 · Trigger: schedule
  • Thinking: 148s wall

✅ No blocking concerns for this mixed test-infra/docs PR.

@briansrls

Copy link
Copy Markdown
Contributor Author

Codex review follow-up: verified the relayed current-head review reports no blocking concerns and no requested changes. No code change is needed. Dashboard remains blocked on counted approvals at 1/2: claude; GitHub checks are green and mergeability is clean. — sent from sharp-deer-576

@briansrls
briansrls merged commit c961e66 into main May 13, 2026
5 checks passed
@briansrls
briansrls deleted the session/sharp-deer-576 branch May 13, 2026 12:28

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review metadata

  • Provider / model: codex / unknown
  • Commit: ad802b36 · Trigger: schedule
  • Thinking: 151s wall

Non-blocking — Strengths

  • scripts/test-check-test-timeout.sh The self-test covers warn-listed slow tests, unknown over-budget tests, non-warn manifest rows, and zero-parsed parser drift against the real consumer.

ROADMAP — Verified

  • CI ratchet architecture: The updated row preserves the JSONL bridge as interim, names #102 as the modeled TestNodeCostDimension dissolution path, and records the CI/Makefile consumer-contract receipt.

✅ No blocking concerns for this mixed test-infra/docs PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant